feat(abtest): hypothesis metadata for pre-registration (Change Set A) - #46

Preserve nested fields in the hypothesis checksum

JSON.stringify applies this array replacer recursively, so nested keys such as type, direction, kind, value, id, and experimentFamilyKey are omitted and each nested object is serialized as {}. Consequently, changing the primary metric, expected lift, owner, or experiment scope after locking still makes verifyHypothesisChecksum return true, defeating the pre-registration integrity guarantee; use deterministic recursive canonicalization that retains nested fields.

Useful? React with 👍 / 👎.

Reject malformed nested launch metadata

Strict validation only requires the nested objects to be present and checks expectedLift.value; it never validates primaryMetric.type/direction, expectedLift.kind or the required absolute-lift unit, and it does not require or validate createdAt. Thus runtime input such as { primaryMetric: {}, expectedLift: { kind: "bogus", value: 1 } } without createdAt passes strict validation and can be locked as a purportedly valid pre-registration.

Useful? React with 👍 / 👎.

Include hypothesis in the operation output schema

The AbTest model now includes hypothesis, but this schema extension only adds assignmentProvenance. When a persisted test contains hypothesis metadata, every CLI/MCP invoker passes the serialized test through parseOperationOutput; Zod strips the undeclared hypothesis property, so callers cannot retrieve the pre-registration even though it remains in storage. Add the hypothesis shape to this shared output schema.

AGENTS.md reference: AGENTS.md:L155-L157

Useful? React with 👍 / 👎.

Validate persisted hypothesis records

Adding hypothesis to the persisted AbTest shape without extending isStoredAbTest means validateStoredAbTestStore and loadStoredAbTests accept arbitrary malformed hypothesis objects and post-lock checksum mismatches, after which parseAbTestStore casts them to HypothesisMetadata. Validate the nested shape and locked-state invariants before hydrating file-backed records so downstream launch/report code does not receive untrusted metadata.

AGENTS.md reference: AGENTS.md:L180-L183

Useful? React with 👍 / 👎.

Validate the supplied lock timestamp before locking

When a caller uses the supported timestamp override with lockHypothesis(metadata, ""), the function returns a checksum with an empty lockedAt; verifyHypothesisChecksum then returns false and a subsequent call is allowed because the double-lock guard tests timestamp truthiness. Any other non-ISO string is also accepted as a valid lock timestamp, so validate the override before creating the locked object.

Useful? React with 👍 / 👎.

Couple absolute-lift units to the primary metric

The independent types permit semantically incompatible metadata such as primaryMetric.type: "click_rate" with an absolute currency_per_recipient lift, or revenue_per_recipient with percentage_point; both combinations also pass validation and can be locked. Model or validate the metric/unit pairing so the pre-registered lift has an interpretable meaning for later analysis and reporting.

Useful? React with 👍 / 👎.

Require non-empty family-key segments

The character-class check accepts delimiter-only and empty-segment keys such as ., -, .foo, foo., and foo..bar. These do not form the documented dotted family identifier and allow a separator typo to create a different collision namespace rather than being rejected; require an alphanumeric segment at each end and between separators.

Useful? React with 👍 / 👎.

Wire hypothesis metadata into test creation

When a test is created through the shared CLI/MCP operation, createAbTestInputSchema, CreateAbTestInput, and AbTestConfig provide no hypothesis field, and AbTestService.createTest provisions the assignment manifest without calling lockHypothesis. Consequently, users of either supported surface cannot pre-register this newly exposed metadata, despite it being returned in operation output; add the hypothesis to the shared creation contract and lock it before segmentation creates the manifest.

AGENTS.md reference: AGENTS.md:L146-L149

Useful? React with 👍 / 👎.

Require hypotheses to be locked when a manifest exists

When loading a record that contains both assignmentManifest and an unlocked hypothesis, these independent optional checks accept it because isStoredHypothesis requires a checksum only if lockedAt is already present. Such a record can therefore acquire or modify hypothesis content after recipient assignment and still pass persistence validation, defeating the stated pre-registration guarantee; enforce that a hypothesis accompanying a manifest has a valid lock and checksum.

Useful? React with 👍 / 👎.

Apply stratified quotas in the assignment path

The new solver is exported but never called by any production code: the holdout provisioning path still invokes segmentSubscribersForHoldout directly, and repository-wide references to computeStratifiedQuotas are limited to this module, its export, tests, and the graph check. Thus A/B tests continue using the existing unstratified manifest regardless of recipient domains, so the released “stratification for A/B test assignment” feature has no operational effect.

Useful? React with 👍 / 👎.

Preserve floor and ceiling bounds during quota correction

When row-wise rounding leaves a column imbalance, accepting any donor with quota > 0 allows repeated swaps to move cells outside the promised floor-or-ceiling allocation. For example, strata {s0:116,s1:105,s2:74,s3:47} and group counts {g0:37,g1:216,g2:63,g3:26} produce s1/g3 = 9 for an ideal of 7.982... (ceiling 8). This can overrepresent a provider stratum even though row and column totals pass; restrict candidate swaps so the donor remains at least its floor and the receiver remains at most its ceiling.

Useful? React with 👍 / 👎.

Validate hypothesis timestamps as ISO 8601

When callers supply audit timestamps, Date.parse accepts non-ISO and normalized-invalid values even though the API promises ISO 8601; for example, createdAt: "0", lockedAt: "01/02/03", and createdAt: "2026-02-30" all pass, with the last silently rolling into March. These ambiguous values are then included in the checksum and accepted by persistence, weakening the pre-registration audit record; use a strict ISO parser such as the datetime schema already used by operation timestamps for both fields.

Useful? React with 👍 / 👎.

Reject malformed nested metadata with validation errors

When this published runtime validator receives metadata from JavaScript or parsed JSON, a present but malformed nested value bypasses the undefined check and is dereferenced directly: primaryMetric: null, expectedLift: null, or experimentScope: null throws a raw TypeError, and owner: {id: 123} does the same at .trim(). Validate that each nested value is an object and each field has the expected primitive type before accessing it so invalid user data consistently produces HypothesisValidationError rather than crashing the caller.

Useful? React with 👍 / 👎.

Document the new public experimentation APIs bilingually

The changesets publish hypothesis locking and recipient-domain stratification as minor user-facing additions, but neither README.md nor README_ko.md documents their contracts, validation rules, or usage, and the package README is unchanged as well. Add matching English and Korean guidance so operators and library consumers can discover and correctly use the newly exported behavior.

AGENTS.md reference: AGENTS.md:L233-L237

Useful? React with 👍 / 👎.

Populate assignment provenance for persisted tests

The new assignmentProvenance field is parsed and serialized but never assigned anywhere in production: deterministic holdout provisioning sets assignmentManifest without setting manifest_v1, while full-split and hydrated legacy tests never receive legacy_unavailable. Consequently, every normal CLI/MCP response omits the field, so consumers cannot use it for its stated purpose of distinguishing deterministic manifests from legacy assignments; derive it when provisioning and when hydrating older records.

Useful? React with 👍 / 👎.

Normalize configured domains before classifying strata

When an operator supplies a custom providerDomainMap containing a semantically valid mixed-case or trailing-dot domain, only the subscriber email is normalized, while domains.includes(domain) compares the configured value verbatim. For example, {google: ["GMAIL.COM"]} classifies user@gmail.com as other, silently skewing the provider quotas; normalize policy domains with the same lowercase, trim, and trailing-dot rules before comparison.

Useful? React with 👍 / 👎.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

minimumStratumSize / smallStratumFallback are declared but never applied.

Nothing in this module merges sub-threshold strata into otherStratumKey, yet StratificationResult.stratumSizes is documented as "Final stratum sizes after small-stratum merge". Callers configuring minimumStratumSize will get tiny strata (e.g. a 1-subscriber kakao row) that defeat the stated purpose of the policy. Either implement the merge before quota computation or drop the fields and the doc claim.

Also applies to: 123-124

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/abtest/src/stratification.ts` around lines 26 - 33, The
stratification flow must apply minimumStratumSize and smallStratumFallback
before computing quotas and final stratumSizes. Update the relevant
stratification and quota-calculation logic to merge every sub-threshold stratum
into otherStratumKey, preserving unknownStratumKey handling and ensuring the
reported sizes reflect the merged result; alternatively remove these
configuration fields and the merge claim if the policy is intentionally
unsupported.

Preserve manifest-only records

When a holdout test is created without the optional hypothesis, or when any pre-change v2 holdout record is loaded, assignmentManifest is present while hypothesis is absent. This new predicate rejects the entire store, so creation can succeed and persist state but the next list/get/write operation fails with test ... failed schema validation. Preserve legacy/no-hypothesis manifests or introduce an explicit schema migration instead of imposing this invariant retroactively.

AGENTS.md reference: AGENTS.md:L180-L183

Useful? React with 👍 / 👎.

Verify pre-locked hypotheses before provisioning

When a library caller supplies HypothesisMetadata with any truthy lockedAt, this branch accepts it without validation or verifyHypothesisChecksum. Because checksum is optional in the public type, a missing or tampered checksum can therefore reach campaign/list provisioning despite the pre-registration integrity guarantee; it may only be rejected on a later store read. Validate the locked timestamp and checksum before performing remote side effects.

Useful? React with 👍 / 👎.

Enable stratification through the production operation

The default policy is disabled, while the sole production caller in AbTestService.createTest passes only { testId }, and neither CreateAbTestInput nor AbTestConfig exposes a stratification policy. Consequently CLI/MCP holdout creation can never enter the new computation and AbTest.stratification always remains undefined; only callers invoking the low-level integration method directly can use the feature.

Useful? React with 👍 / 👎.

Apply the configured small-stratum fallback

For an enabled policy using the default threshold, a provider represented by fewer than 20 recipients is still retained as its own row. Neither classifyStratum, computeStratifiedQuotas, nor the integration tally reads minimumStratumSize or smallStratumFallback, so these public policy fields have no effect and the returned stratumSizes contradicts the documented merge-into-other behavior.

Useful? React with 👍 / 👎.

Keep quota swaps within both cell bounds

When no preferred bounded swap is found, the donor check only requires a positive quota, allowing correction to reduce a cell below floor(ideal) and violate the solver's stated floor-or-ceiling contract. For example, strata {s0:90,s1:420,s2:13,s3:34,s4:23,s5:90} and groups {g0:126,g1:414,g2:130} return s2/g1 = 7 for an ideal of about 8.033, even though a bounded allocation exists. Use a globally valid bounded correction rather than accepting an out-of-band donor.

Useful? React with 👍 / 👎.

Add hypothesis input to the CLI adapter

Although the shared create operation and MCP schema now accept hypothesis, the inspected CLI create path in apps/cli/src/commands/abtest.ts defines no hypothesis option and buildCreateInputFromFlags cannot populate this field. CLI users therefore cannot use the new pre-registration behavior while MCP callers can, leaving the two advertised surfaces with different creation contracts.

AGENTS.md reference: AGENTS.md:L146-L149

Useful? React with 👍 / 👎.

Reject an all-zero quota matrix

When totalAudience, every stratum size, and every group count are zero, both equality checks pass and the solver divides 0 * 0 by zero. It then returns cells containing NaN quotas and ideals instead of rejecting the empty input, which cannot be safely serialized or used for validation. Explicitly reject a non-positive totalAudience before calculating ideals.

Useful? React with 👍 / 👎.

Update both root user guides

This change adds user-visible hypothesis input and stratification output but documents them only in the package README; the repository-wide English and Korean guides remain unchanged even though they contain the CLI/MCP A/B-test usage sections. Add corresponding guidance to both README.md and README_ko.md so the new behavior is discoverable consistently.

AGENTS.md reference: AGENTS.md:L233-L237

Useful? React with 👍 / 👎.

Build the provider lookup once per audience

When stratification is enabled for a large audience, every call to classifyStratum rebuilds and normalizes the complete provider-domain map. The integration calls this once per recipient, making the supposedly single-pass tally perform O(recipients × configured domains) work and allocate a new Map for every subscriber. Build the lookup once per policy and reuse it throughout the audience classification pass.

Useful? React with 👍 / 👎.

Validate persisted stratification data

The new stratification field has an operation-output schema but no corresponding check in isStoredAbTest. A state file containing stratification: "bad", negative quotas, or malformed cells therefore passes loadStoredAbTests, is cast to AbTest, and only fails later when list/get output parsing encounters the invalid value. Add a structural persistence guard for the complete quota result so corrupt state is rejected at the file boundary.

AGENTS.md reference: AGENTS.md:L180-L183

Useful? React with 👍 / 👎.

Apply stratified quotas to the actual assignments

When segmentSubscribersForHoldout is called with an enabled stratification policy, the holdout and variant lists have already been populated from the globally ranked slices before this matrix is computed. Consequently the returned matrix is only a target: it cannot stop one provider from being concentrated in a variant, and AbTest.stratification can report balanced quotas that do not describe the actual list memberships. Apply the quotas while constructing the subscriber slices, before any bulk list updates.

Useful? React with 👍 / 👎.

Make quota correction converge for valid margins

The greedy selection of the first deficit and first surplus can consume the only viable swap for a later pair and then fail even though the row and column margins are valid. For example, stratumSizes={s0:4,s1:2,s2:2,s3:4} and groupExactCounts={g0:1,g1:4,g2:4,g3:3} with audience 12 throws column "g0" has residual deficit -1. In provisioning this exception is swallowed and stratification silently disappears, so the correction needs a convergent allocation/backtracking strategy rather than fixed first-pair swaps.

Useful? React with 👍 / 👎.

Validate pre-locked hypotheses before provisioning

For direct callers of the exported AbTestService, this branch checks only that the checksum matches; it never runs strict metadata validation. A caller can therefore supply a correctly checksummed object with an empty objective, negative lift, malformed timestamps, invalid metric/family key, or empty owner ID and have it accepted and provisioned, whereas the unlocked path rejects the same data through lockHypothesis. Run strict validation, including the lock timestamp, before accepting a pre-locked hypothesis.

Useful? React with 👍 / 👎.

Enforce stratification invariants when loading state

Despite the guard's contract, it returns true after checking only individual numeric shapes and never verifies that quota rows correspond to stratumSizes, that cells reference existing row/group keys, that cell quotas match quotas, or that row sums equal their stored sizes. Thus a record such as quotas: {gmail:{A:1}}, stratumSizes:{gmail:100}, cells:[] is accepted and exposed by get/list operations as valid stratification data. Validate these cross-field invariants before accepting persisted state.

Useful? React with 👍 / 👎.

Bind the hypothesis checksum to the assignment manifest

For a persisted test that already has an assignment manifest, this condition accepts any independently valid locked hypothesis; it does not compare the hypothesis checksum with provenance captured when the manifest was created. Replacing the hypothesis with a newly locked post-hoc hypothesis therefore passes loading without discarding or rebuilding the existing assignment, defeating the stated pre-registration guarantee. Persist the original hypothesis checksum with the assignment provenance or manifest and require an exact match here.

Useful? React with 👍 / 👎.